In order to understand the context of certain embodiments of the invention, the following provides an explanation of certain technical aspects of a packet based telecommunications network environment. Internet/Intranet technology is based largely on the TCP/IP protocol suite. At the network level, IP provides a “datagram” delivery service—that is, IP is a protocol allowing for delivery of a datagram or packet between two hosts. By contrast, TCP provides a transport level service on top of the datagram service allowing for guaranteed delivery of a byte stream between two IP hosts. In other words, TCP is responsible for ensuring at the transmitting host that message data is divided into packets to be sent, and for reassembling, at the receiving host, the packets back into the complete message.
TCP has “flow control” mechanisms operative at the end stations only to limit the rate at which a TCP endpoint will emit data, but it does not employ explicit data rate control. The basic flow control mechanism is a “sliding window”, a window which by its sliding operation essentially limits the amount of unacknowledged transmit data that a transmitter is allowed to emit. Another flow control mechanism is a congestion window, which is a refinement of the sliding window scheme involving a conservative expansion to make use of the full, allowable window.
The sliding window flow control mechanism works in conjunction with the Retransmit Timeout Mechanism (RTO), which is a timeout to prompt a retransmission of unacknowledged data. The timeout length is based on a running average of the Round Trip Time (RTT) for acknowledgment receipt, i.e. if an acknowledgment is not received within (typically) the smoothed RTT+4*mean deviation, then packet loss is inferred and the data pending acknowledgment is re-transmitted. Data rate flow control mechanisms which are operative end-to-end without explicit data rate control draw a strong inference of congestion from packet loss (inferred, typically, by RTO). TCP end systems, for example, will “back-off,”—i.e., inhibit transmission in increasing multiples of the base RTT average as a reaction to consecutive packet loss.
A crude form of bandwidth management in TCP/IP networks (that is, policies operable to allocate available bandwidth from a single logical link to network flows) is accomplished by a combination of TCP end systems and routers which queue packets and discard packets when some congestion threshold is exceeded. The discarded and therefore unacknowledged packet serves as a feedback mechanism to the TCP transmitter. Routers support various queuing options to provide for some level of bandwidth management. These options generally provide a rough ability to partition and prioritize separate classes of traffic. However, configuring these queuing options with any precision or without side effects is in fact very difficult, and in some cases, not possible. Seemingly simple things, such as the length of the queue, have a profound effect on traffic characteristics. Discarding packets as a feedback mechanism to TCP end systems may cause large, uneven delays perceptible to interactive users. Moreover, while routers can slow down inbound network traffic by dropping packets as a feedback mechanism to a TCP transmitter, this method often results in retransmission of data packets, wasting network traffic and, especially, inbound capacity of a WAN link. In addition, routers can only explicitly control outbound traffic and cannot prevent inbound traffic from over-utilizing a WAN link. A 5% load or less on outbound traffic can correspond to a 100% load on inbound traffic, due to the typical imbalance between an outbound stream of acknowledgments and an inbound stream of data.
In response, certain data flow rate control mechanisms have been developed to provide a means to control and optimize efficiency of data transfer as well as allocate available bandwidth among a variety of business enterprise functionalities. For example, U.S. Pat. No. 6,038,216 discloses a method for explicit data rate control in a packet-based network environment without data rate supervision. Data rate control directly moderates the rate of data transmission from a sending host, resulting in just-in-time data transmission to control inbound traffic and reduce the inefficiencies associated with dropped packets. Bandwidth management devices allow for explicit data rate control for flows associated with a particular traffic classification. For example, U.S. Pat. No. 6,412,000, above, discloses automatic classification of network traffic for use in connection with bandwidth allocation mechanisms. U.S. Pat. No. 6,046,980 discloses systems and methods allowing for application layer control of bandwidth utilization in packet-based computer networks. For example, bandwidth management devices allow network administrators to specify policies operative to control and/or prioritize the bandwidth allocated to individual data flows according to traffic classifications. In addition, certain bandwidth management devices, as well as certain routers, allow network administrators to specify aggregate bandwidth utilization controls to divide available bandwidth into partitions. With some network devices, these partitions can be configured to ensure a minimum bandwidth and/or cap bandwidth as to a particular class of traffic. An administrator specifies a traffic class (such as FTP data, or data flows involving a specific user) and the size of the reserved virtual link—i.e., minimum guaranteed bandwidth and/or maximum bandwidth. Such partitions can be applied on a per-application basis (protecting and/or capping bandwidth for all traffic associated with an application) or a per-user basis (controlling, prioritizing, protecting and/or capping bandwidth for a particular user). In addition, certain bandwidth management devices allow administrators to define a partition hierarchy by configuring one or more partitions dividing the access link and further dividing the parent partitions into one or more child partitions.
While the systems and methods discussed above that allow for traffic classification and application of bandwidth utilization controls on a per-traffic-classification basis operate effectively for their intended purposes, they possess certain limitations. As discussed more fully below, identification of traffic types associated with data flows traversing an access link involves the application of matching criteria or rules to various characteristics of the data flows. Such matching criteria can include source and destination IP addresses, port numbers, MIME types, etc. After identification of a traffic type corresponding to a data flow, a bandwidth management device associates and subsequently applies bandwidth utilization controls (e.g., a policy or partition) to the data flow corresponding to the identified traffic classification or type. A common use of bandwidth management devices is to limit the bandwidth being consumed by unruly, bandwidth-intensive applications, such as peer-to-peer applications (e.g., Kazaa, Napster, etc.). Network savvy users (such as students in a campus or university environment), however, have become aware that such bandwidth management devices have been deployed to limit or restrict such unauthorized network traffic. As a result, users often attempt to bypass or thwart the bandwidth management scheme effected by such bandwidth management devices by creating communications tunnels (proxy tunnels) through which unauthorized or restricted network traffic is sent. The attributes discernible from the content of these tunneled data flows, however, often reveal little information about its true nature. For example, commercial HTTP tunnel services (such as loopholesoftware.com, TotalRc.net, and http-tunnel.com, etc.) allow users to send all network traffic in the form of HTTP traffic through a HTTP tunnel between a tunnel client and an HTTP proxy server maintained by the tunnel services provider. FIG. 6 illustrates the functionality and operation of a typical HTTP proxy tunnel. Client device 42 includes a client application (such as a peer-to-peer application 71) and a tunnel client 72. The client application sends data to the tunnel client 72 which tunnels the data over HTTP to a tunnel proxy server 74. The tunnel proxy server 74 then forwards the data to the intended destination (here, network resource 75), and vice versa. Such HTTP tunnels typically feature encryption; accordingly, a bandwidth management device 30, encountering the tunneled traffic in this form, may not detect the exact nature of the traffic and, in fact, classify such data flows as legitimate or regular HTTP traffic. Accordingly, these tunneling mechanisms and other techniques for evading bandwidth utilization controls implemented by bandwidth management devices present new challenges to network administrators and bandwidth device manufacturers desiring to effectively control unauthorized or restricted network traffic.
In light of the foregoing, a need in the art exists for methods, apparatuses and systems allowing for bandwidth management schemes that are responsive to the utilization characteristics associated with individual users. A need in the art further exists for methods, apparatuses and systems allowing for detection of questionable or other activities designed to evade bandwidth management control schemes and, thus, enabling application of more rigorous network traffic classification mechanisms and/or disparate bandwidth utilization controls. Embodiments of the present invention substantially fulfill these needs.